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1  INTRODUCTION 


To  vajying  degrees,  all  tests  of  ability  contain  items  that  violate  the  assumption 
of  conditional  independence  on  which  much  of  present  item  response  theory  (IRT) 
is  based.  If  the  association  between  pairs  of  items  cannot  be  explained  entirely  by 
their  relationship  with  the  underlying  ability,  conventional  methods  for  evaluating 
the  likelihood  of  latent  structure  models  become  inappropriate  and  may  produce 
biased  estimates  of  item-parameters  and  corresponding  estimates  of  ability.  Under 
the  assumption  of  conditional  independence,  the  probability  of  subject  i  responding 
in  pattern  a:;  =  [T,i,Xi2 . •Tin]'  conditional  on  ability  &,  is 


p{x  =  X.  1 9,)  =  (u 

J=1 

where  is  the  jth  item  score  for  subject  i  (x.y  =  1  if  correct,  otherwise  =  0), 
and  P}(9)  is  the  item  response  function  (IRF)  that  e.'presses  the  probability  that  a 
subject  with  ability  9,  will  respond  correctly  to  item  j.  The  IRF  may  be  obtained 
from  any  one  of  several  item-response  models  (e.g.,  one,  two,  or  three  parameter 
logistic  or  normal  ogive  models).  If  the  ability  9  under  study  does  not  account  for 
all  of  the  association  between  items,  the  assumption  of  conditional  independence  is 
not  valid  and  the  conditional  probability  cannot  be  expressed  simply  as  a  continued 
product  of  the  individual  item-response  probabilities. 

Three  general  approaches  to  the  problem  of  failure  of  conditional  independence 
are  available.  First,  we  can  simply  ignore  the  dependence  and  risk  obtaining  bias  in 
our  estimates  of  item  parameters  and  ability  and  corresponding  estimates  of  preci¬ 
sion.  Stout  ( 1987)  has  shown  that  if,  for  a  fixed  level  of  ability  9,  the  average  item 
covariance  is  "negligible'’  as  test  length  grows,  that  is, 

I  n  a 

limn-oo— ^^cov(i,,Xj  I  ^)  —  0,  (’2) 

^  .=ij=i 

then  the  test  is  “essentially  independent"  and  the  usual  results  for  the  locally  inde¬ 
pendent  case  apply  (i.e.,  equation  1).  As  rigorous  tests  for  essential  independence 
become  available,  this  will  provide  the  means  for  judging  one’s  confidence  in  selecting 
this  alternative. 

Second,  if  we  saturate  the  latent  space  with  additional  dimensions,  we  can  at  some 
point  achieve  a  conditionally  independent  solution.  If  the  required  dimensionality 
is  small,  say  2  or  3,  this  may  be  feasible  using  the  approach  described  by  Bock 
and  Aitkin  (1981),  where  the  integration  of  the  Jf-dimensional  ability  distribution 
g(0)  ~  iV(0, 1)  is  approximated  using  Gaussian  quadrature  formulae.  There  are  two 
potential  drawbacks  to  this  approach.  First,  if  there  are  numerous  “method”  related 
factors  and  a  single  primary  ability  factor,  the  dimensionality  required  to  bring  about 
conditional  independence  may  be  too  high  for  practical  purposes.  Second,  if  there 
is  a  single  method  related  factor  and  i  single  primary  ability,  it  may  be  difficult  to 
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find  a  two-dimensional  solution  that  preserves  this  structure.  Indeed,  the  orthogonal 
solution  will  often  divide  the  items  based  on  the  method  related  factor. 

The  third  approach  to  this  problem  is  to  formulate  a  generalized  IRT  model 
of  dependence  in  which  the  simplicity  of  a  unidimensional  model  is  retained.  In 
the  generalized  model  the  conditional  response-pattern  probability  is  of  the  form 
P{x  =  I,  I  where  E  is  an  n  X  n  symmetric  inter-item  covariance  matrix.  It 

should  be  clear  that  P{x  =  Xi\  E)  is  equcd  to  the  right  side  of  (1)  only  when  E  is 
diagonal,  which  is  only  true  under  conditional  independence.  When  E  has  nonzero 
off-diagonal  elements.  P{x  =  Xi  \  E)  becomes  difficult  to  evaluate,  and,  in  fact, 
no  closed  form  expression  has  been  obtained  for  general  S  beyond  n  =  3.  The 
result  of  our  preliminary  studies  has  been  the  development  of  an  approximation  of 
P(x  =  I,  I  E)  for  general  E  ajid  n  <  50.  The  focus  of  our  current  work  is  on  the 
development  of  a  unidimensional  IRT  model  of  dependence  that  incorporates  these 
generalized  probability  estimates. 

2  THEORETICAL  BASIS  FOR  MODELS  OF  DEPENDENCE 

Experimentally,  failure  of  conditional  independence  can  result  from  two  general  con¬ 
ditions:  item  content  and  item  presentation. 

2.1  Item  Content 

If  different  items  require  different  abilities,  aptitudes  or  cognitive  processes,  then  the 
IRF  must  depend  not  only  on  the  primary  ability,  but  on  the  secondary  cognitive 
skills  or  method  related  factors  as  well.  As  a  result,  association  among  the  residuals 
is  introduced  and  the  assumption  of  conditional  independence  of  a  unidimensional 
IRT  model  is  no  longer  tenable. 

For  example,  assume  that  the  ability  space  is  two-dimensional,  where  repre¬ 
sents  a  primary  ability  and  62  ^  secondary  skill  that  is  required  for  success  on  some 
of  the  items.  With  respect  to  item  j,  the  underlying  response  process  (response- 
strength  variable)  is: 


yj  -  +  ^j2^2  +  5_, 

Conditional  on  9i,  the  variance  of  yj  is: 


(3) 


=  v(5, -hAjjOj) 

=  V(5,)-hV(A,2<?2) 

=  1-Aj, 

= 

(4) 


This  result  follows  because  V(9i)  =  V(^2)  =  1.  Conditional  on  Oi,  the  residual 
inter- item  covariances  are: 


for 
.  I 


1  oa 


v  on/ 

Availability  Codes 
i  jAvull  p.nd/or 

Dial  Siooclol 
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□  □ 


cov{yi,yj  I  ^i)  =  A,2Aj2 
or  expressed  as  a  residual  correlation, 


(5) 


_ A.2Aj2 _ 

“  Ml 


(6) 


2.2  Item  Presentation 

In  terms  of  item  presentation,  conditional  independence  implies,  that  for  a  fixed 
level  of  ability,  the  probability  of  a  correct  response  to  item  j  is  independent  of  the 
examinee’s  performance  on  all  other  items.  However,  in  the  context  of  adaptive 
testing,  the  presentation  of  item  j  +  1,  is  in  fact,  conditional  on  the  success  or 
failure  on  item  j,  where  the  items  1,2, ....j  +  1  are  ordered  in  terms  of  difficulty, 
presumably  along  a  single  ability  dimension  of  interest.  In  practical  terms,  this 
assumption  implies  that  the  individual  item  response  probabilities  for  subjects  with 
the  same  ability  would  be  identical  for  all  possible  orderings  of  the  items  on  the  test. 

When  item  presentation  is  the  result  of  a  random  process,  the  assumption  of  order 
invariance  may  well  be  reasonable.  However,  when  item  presentation  is  systematic 
and  based  on  either  progressive  linear  increases  or  decreases  in  difficulty  (conditional 
on  a  provisional  estimate  of  examinee  ability),  it  seems  likely  that  the  conditional 
item-response  probabilities  would  be  affected.  For  example,  if  the  initial  ability 
estimate  is  low,  several  items  of  increasing  difficulty  may  be  presented.  If  a  sequence 
of  similar  problems  is  presented,  the  probability  of  a  correct  response  may  be  both 
a  function  of  the  e.xaminee’s  primary  ability  and  in  addition  the  subject’s  skill  at 
recognizing  the  sequence,  level  of  expectation,  and  perhaps  even  learning. 

.A.t  the  very  least,  we  must  admit  the  possibility  that  random  orderings  and 
systematic  orderings  of  items  have  some  impact  on  the  response  process  and  cor¬ 
responding  probability  estimates.  As  an  analogy,  consider  an  auditory  stimulation 
experiment  in  which  the  lower  bound  of  the  subject’s  hearing  level  is  to  be  estab¬ 
lished.  If  the  volume  of  the  tone  is  presented  in  uniform  steps  from  loud  to  soft, 
the  resulting  estimate  of  hearing  level  and  corresponding  estimate  of  precision  (if 
replicated)  will  clearly  be  different  than  if  the  volume  of  the  tone  is  presented  in  a 
random  order.  Clearly,  the  knowledge  of  the  presentation  pattern  and  expectation 
of  the  degree  of  difficulty  of  the  next  discrimination  task  (i.e.,  tone  versus  noise)  will 
contribute  to  the  likelihood  that  the  subject  will  make  the  correct  response.  .\s  will 
be  demonstrated,  the  consequence  of  ignoring  this  residual  association  will  primarily 
be  an  underestimate  of  the  posterior  standard  deviation.  Since  the  convergence  of 
an  adaptive  testing  session  is  often  based  on  the  precision  of  the  estimated  ability, 
on  average,  the  result  will  be  a  premature  conclusion  of  the  testing  session,  and  false 
sense  of  certainty. 
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3  Approximating  Multivariate  Normal  Orthant  Probabilities 

In  this  section,  we  describe  a  general  approximation  for  multivariate  normal  orthant 
probabilities.  Using  a  threshold  argument,  we  show  how  the  approximation  can  be 
used  to  estimate  the  probability  of  any  of  the  2”  possible  binary  response  patterns 
realized  in  a  testing  session. 


3.1  The  Clark  Algorithm 

Based  on  earlier  work  by  Clark  (1961),  Gibbons,  Bock  and  Hedeker  (1987)  developed 
a  very  general  approximation  to  the  probability  that  n  correlated  random  variables 
drawn  from  a  multivariate  normal  distribution  are  jointly  non-negative;  that  is; 

/  /  •■•  /  ^n(.ri,-C2 . x„}dxi,dx2,...,dx„.  (7) 

Jo  Jo  Jo 

Assuming  that  the  underlying  response  process  is  multivariate  normal  and  that 
the  correct  responses  to  a  series  of  items  on  a  test  axe  the  result  of  exceeding  a 
threshold  on  one  or  more  "latent”  continua  0,  our  modified  Clark  algorithm  can 
provide  probability  estimates  of  any  of  the  2"  possible  binary  response  patterns 
without  any  restriction  on  the  form  of  the  inter-item  covariances  cifter  conditioning 
on  8.  .\n  overview  of  Clark's  original  formulae  and  our  modification  of  the  algorithm 
is  now  presented. 

Designating  positive  directions  1  and  negative  directions  0.  we  may  represent 
the  probability  of  the  positive  orthant  of  an  n- variate  distribution  by  P(l,  1, . . . ,  1), 

that  of  the  negative  orthant  by  P(0,0 _ ,0),  and  that  of  any  one  of  the  other 

2''  -  2  orthants  by  inserting  the  appropriate  pattern  of  I’s  and  O’s.  The  Clark 
algorithm  provides  a  computing  appro,ximate  for  any  orthant  of  a  multivariate  normal 
distribution  with  arbitrary  vector  mean  and  covariance  matrix.  Clark  (1961)  derives 
the  following  formulas. 

Let  any  three  successive  components  from  an  n- variate  vector,  j/j,  be  distributed: 


1 

~.vf 

fit 

\ 

yi+i 

di+i 

1 

^i+l  Pi,i+l 

V 

.  ^‘+2  . 

_  O’,  <7,  +  2  Pi.i-t-Z  +  2  Pi+l.t  +  2 

_2 

y 

Let  jli  =  max(j/,)  =  y,,  and  compute  the  probability  that  y,>i  >  i/i  as  follows: 


set 

^i+i 

=  (/l.  )/C<-H 

where 

C.>I 

=  ' 

Then 

Plyi+i  >  y) 

=  Piyi+i  -  y>0) 
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the  value  of  the  univariate  normal  distribution  function  at  the  standard  deviate 

-t  +  l  • 

Now  let  yi+i  =  max(yj,  j/i+i)  and  assume  (as  an  approximation)  that  ( j/,+2- iii+i ) 
is  bivariate  normal  with  means, 

^(yi+i)  ~  ^{yi+2)  —  Mi+2  (10) 

M(y,+i)  =  f(y,+i)  =  +  /^.+i^(-^t+i)  +  C.+id>(^.+i), 

variances 


where 

y?+i)  =  *1+1  )+(Mi>i+<^i>i)^(~'i+i  )+(Mt+Mt+i  )Ci+i<?(*i+i  )i(  ii) 


and  correlation 

p{yx+i,yi+2) 


Then, 


t^i72i.i+2^(^t^-l )  +  0'i+lPi+l,i+2^(~^t+l) 

<^(yi+i) 


(12) 


P{yi+2  =  max(j/i,y,+i,y,+2))  =  ^((y.+2  -  yi+i  >  0)  n  (yi+2  -  yi  >  0))  (13) 
is  approximated  by 

^*(y.+2  >  y.+i )  =  P{yi+2  -  y.+i  >  o) 

_ Pt+2  -  p(yi+\) _ 

\/<^?+2  +  <7^(y.>i)  -  2(7, +2«^(yi+i)p(yi+i.  yi+2) 

.\ssuming  as  a  working  approximation  that  i/i+i  is  normally  distributed  with 
the  above  mean  cind  variance,  we  may  therefore  proceed,  recursively  from  i  =  1  to 
(  =  n  —  1,  where  yn+i  is  an  independent  dummy  variate  with  mean  zero  and  variance 
zero  (i.e.  y„+i  =  0).  Then, 


^’[yn+i  =  max(i/i.y2 . yn+i)] 

=  F[(yn+i  -  yi  >  0)n  (y„+i  -  ^2  >  0)n  . . . n  (y„+i  -  y„  >  0)] 

=  F[(-y,  >0)n(-y2>0)n...n(-y„>0)] 

approximates  the  probability  of  the  negative  orthant  of  the  specified  multivariate 
normal  distribution.  The  probability  of  any  other  orthant  can  be  obtained  by  re¬ 
versing  the  signs  of  the  variates  corresponding  to  I’s  in  the  orthant  pattern. 


5 


3.2  The  Modified  Clark  Algorithm 

In  an  earlier  paper  Gibbons  and  Bock  (1987)  noted  that  the  accuracy  of  the  Clark 
approximation  diminishes  with  increasing  magnitude  of  the  correlations.  If  we  ap¬ 
ply  the  Clark  approximation  directly  to  estimates  of  inter-item  correlations,  it  will 
generally  yield  biased  results  due  to  the  size  of  correlations.  This  is  true  regardless 
of  whether  the  correlation  matrix  exhibits  the  property  of  conditional  independence. 
Alternatively,  if  we  examine  the  residual  inter-item  correlation  matrix  at  fixed  points 
on  the  ability  scale,  we  will  observe  the  identity  matrix  for  conditionally  independent 
solutions  or  small  residual  correlations  for  those  item  pairs  that  are  conditionally  de¬ 
pendent.  In  light  of  this,  we  evaluate  the  response  function  at  several  fixed  points 
on  the  ability  scale  using  Gauss- Hermite  quadrature,  and  correct  these  estimates 
using  the  Clark  algorithm.  These  corrections  depend  only  on  the  residual  inter-item 
correlations,  which  in  practice  should  be  quite  small.  The  modified  Clark  algorithm 
proceeds  as  follows. 

Step  1  Obtain  a  factor  solution  of  dimension  A^,  using  full  information  factor  analysis 
for  binary  data  (Bock  and  Aitkin,  1981;  Bock,  Gibbons  and  Muraki.  1988). 

Step  2  Using  the  estimated  factor  loadings  for  dimensions  2 . . .  A',  compute  the  esti¬ 
mated  residual  correlation  matrix  R’;  that  is,  the  residual  correlations  among  the  n 
items  conditioning  on  the  primary  ability  dimension  6i.  For  example,  when  K  =  2. 


,  _  ^i2^j2 _ 


(14) 


Step  S  Given  the  previous  values  of  item  thresholds  n  amd  primary  item  factor 
loadings  Aji  compute  the  invariant  item  pwameters  aj  (slope)  and  cj  (intercept)  as: 


and 

Cj  =  -Tj/'/l  - 


Step  5  .\t  each  point  on  the  ability  dimension  (i.e.  at  each  quadrature  node  .V;.  i 
comijuto  the  value  of  tlie  re.si>onse  function  for  each  item  as: 

=  Cj  +  ajA'fc  (15) 

where  Xk  are  the  nodes  of  the  Gauss- Hermite  polynomial  (see  Stroud  and  Sechrest, 

1966). 

Step  6  At  each  quadrature  point,  substitute  the  values  of  Zjk  for  the  mean  vector  n 
and  R*  for  the  covariance  matrix  E  and  compute  the  Clark  approximated  probability 
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Ci(Xk)-  Accumulating  these  probabilities  over  all  quadrature  nodes  for  a  given 
response  pattern  (xi)  yields  the  desired  marginal  probability  estimate 


h{x,)  =  r 

«/— oo 

fc=i 

where  A(Jffc)  is  the  corresponding  weight  at  quadrature  node  Xk- 

We  note  that,  in  practice,  the  effect  of  assuming  normality  of  the  maximum  of 
two  jointly  normal  variables,  overestimates  the  probability  in  the  tail  of  the  distribu¬ 
tion.  To  correct  this,  we  apply  an  empirically  based  correction  factor  to  probability 
estimates  that  are  less  than  0.04; 

CiiXk)  =  C;(  ( 16) 

This  correction  factor  has  been  found  to  provide  the  necessary  adjustment  across 
the  entire  quadrature  space  as  tested  on  tests  of  length  5  to  40  items. 


3.3  Simulation  Studies 

To  simulate  the  desired  response  process,  we  began  by  constructing  a  two-dimensional 
factor  structure  that  might  be  encountered  in  a  typical  testing  situation.  For  the 
five-item  case,  we  selected  the  factor  matrix: 


A  = 


.7 

.6 

.5 

.4 

.3 


0 

0 

0 

0 

0 


(17) 


Assuming  a  correlation  between  the  two  ability  dimensions  of  .5,  the  orthogonal 
projection  of  this  matrix  (i.e.,  A(T“^)'  where  T  is  the  Cholesky  factor  of  the  9.x2 
correlation  matrix),  is: 


.70 

-.40 

.60 

-..15 

.50 

-.29 

.40 

-.23 

.30 

-.17 

For  n  =  10,20,  and  40,  the  matrix  was  replicated  2,  4  and  8  times  respectively. 
Given  this  factor  structure,  the  item-responses  were  simulated  as  follows. 

Step  1:  Obtain  n  -F  2  random  normal  deviates.  Call  the  first  two  zi  and  Z2  and  the 
remaining  n,  ci , . . . , 
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Step  2:  Simulate  the  primary  ability  of  subject  t  (fii)  as  Zi  =  (zi+ff)(rg,  where  0  and 
(Tg  are  the  mean  and  standard  deviation  of  the  generating  ability  distribution  [i.c., 
N(9,a^)].  For  studies  of  EAP  ability  estimates,  zi  is  generally  fixed  at  0.  1.  or  2. 

Step  ■]:  Determine  the  secondary  ability  of  subject  i  [O2)  as  zj  .V(0.  1). 

Step  4‘  Compute  the  residual  disturbance  for  each  item  as  fj  =  CjOj,  where  Oj  = 
^1  -  Aji  -  Ajj. 

Step  5:  Compute  the  response  process  for  item  j  as: 


Vi  —  +  Aj222  +  (j 


(19) 


(Note:  for  conditional  independence  Xj2  =  0). 

Step  6:  If  yj  >  then  the  response  to  item  j  for  subject  i  is  correct  {x,j  =  1) 
otherwise  it  is  incorrect  (x,^  =  0). 

3.4  Method  for  Study  No.  1 

Using  the  previously  described  method  of  simulating  item-response  patterns,  wc 
computed  the  accuracy  of  several  methods  of  computing  item-response  pattern  prob¬ 
abilities.  These  methods  included  the  original  Clark  algorithm,  the  modified  Clark 
algorithm,  the  modified  Clark  algorithm  with  R*  =  I,  one-dimensional  Gaussian 
quadrature  (40  points),  and  two-dimensional  Gaussian  quadrature  (1600  points). 
We  would  expect  similar  results  for  the  modified  Clark  algorithm  and  the  two- 
dimensional  Gaussian  quadrature,  and  similar  results  for  the  modified  Clark  algo¬ 
rithm  with  R*  =  I  and  one-dimensional  quadrature.  The  original  Clark  algorithm 
should  perform  poorly  throughout.  The  simulation  study  consisted  of  four  condi¬ 
tions;  5  and  10  items  each  with  conditional  independence  and  dependence.  The 
accuracy  of  these  five  probability  estimators  was  evaluated  by  comparing  them  to 
the  corresponding  Monte  Carlo  estimates  obtained  from  one  million  simulates  for  5 
items  and  ten  million  for  10  items. 

3.5  Results 

The  results  of  the  first  simulation  study  are  displayed  in  Tables  1-4.  Tables  1  and  d 
display  the  probability  estimates  for  the  5  and  10  item  conditional  iiulepondeiK c  ca.sc' 
and  Tables  2  and  4  display  results  for  the  case  of  dependence.  For  indeieMidence.  on<- 
and  two-dimensional  quadrature  results  are  identical  as  are  the  two  modified  Clark 
algorithms.  For  five  items  and  independence  (Table  1),  the  results  are  virtually 
identical  for  modified  Clark  and  quadrature  methods;  whereas  the  original  Clark 
performed  poorly  as  expected.  For  10  items  and  independence  (Table  3),  the  modified 
Clark  algorithm  actually  out  performed  the  quadrature  for  the  14  selected  patterns 
and  did  slightly  worse  for  the  total  number  of  patterns  realized  in  the  sample  of  ten 


8 


million.  Again,  the  original  Clark  algorithm  performed  poorly  relative  to  the  other 
estimators. 

For  conditional  dependence,  on  five  items  (Table  2),  the  modified  Clark  and 
two-dimensional  (juadrature  estimates  were  virtually  identical.  Similarly,  when  the 
residuals  are  incorrectly  assumed  to  be  zero  (i.e.,  R’  =  I),  the  modified  Clark  and 
one-dimensional  quadrature  yielded  virtually  identical  results  as  expected.  Again, 
the  original  Clark  performed  poorly.  For  10  items  (Table  4)  similar  results  were 
obtained. 

4  Estimating  Ability  in  the  Presence  of  Conditional  Dependence 

Our  next  study  was  designed  to  determine  the  impact  of  conditional  dependence 
on  estimates  of  ability.  Again,  we  were  concerned  with  the  case  in  which  both  a 
primary  ability  dimension  and  a  method  related  dimension  were  both  present  in 
tests  of  variable  length,  administered  to  samples  of  varying  ability. 

4.1  EAP  estimates  of  Ability 

Let  z,j  =  1  if  item  j  is  answered  correctly  by  respondent  i.  otherwise  x,j  =  0.  The 
probability  that  item  j  is  answered  correctly  by  a  respondent  with  ability  9  is: 


F(xj  =  i\e)  =  ^j(e) 


(20) 


The  likelihood  of  6,  given  the  response  vector 
—  [®1 1  •  •  •  ^nji 


(21) 


is 

-  4»,(»)]‘-*^  (22) 

j=i 


The  EAP  estimate  of  the  ability  of  subject  i.  0,.  given  the  item  responses  x,.  is 
approximated  as 


EU,  XkL.iXkm-'iXk) 


and  the  posterior  standard  deviation  is 


1 2:5 1 


PSD(ffi)  = 


ELxiXk  -  i?L,(XkW{Xk) 
ELi  L,{Xk)W{Xk) 


(24) 


where  Xk  is  one  of  the  q  quadrature  nodes  and  VF(Afc)  is  the  corresponding  normal¬ 
ized  quadrature  weight. 
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4.2  EAP  Ability  Estimates  via  the  Clark  Algorithm 

When  the  assumption  of  conditional  independence  is  violated,  the  previous  estimator 
is  no  longer  valid,  because  the  continued-product  probability  requires  all  inter-itcm 
residual  covariances  to  be  zero.  If  responses  to  a  particular  test  are  determined 

both  by  a  primary  ability  and  by  a  method  related  dimension,  the  residual  inter¬ 
item  covariances  will  be  nonzero  even  if  the  primary  ability  and  method  related 
dimensions  are  uncorrelated.  In  this  case,  the  modified  Clark  algorithm  may  be 
used  to  obtain  the  correct  likelihood,  C'(Xk),  which  can  be  substituted  for  Li(Xk) 
in  the  previous  equations,  and  correct  estimates  of  the  mean  and  variance  of  the 
posterior  distribution  can  be  obtained. 

4.3  Method  for  Study  No.  2 

The  conditions  of  the  experiment  included: 
number  of  items  (10,  20  and  40) 
level  of  ability  -  ^  =  0, 1,2 
and  conditional  dependence 

In  all  cases,  40  quadrature  points  were  selected  to  fill  a  ±2  range  about  the  generating 
ability  value.  For  the  two-dimensional  quadrature  solution,  the  second  dimension 
was  evaluated  using  40  quadrature  points  between  -2  and  2,  since  ^2  ~  JV{0.1). 
Item  thresholds  were  selected  as  equally  spaced  points  between  -2  and  2.  For 
each  condition,  500  examinees  were  simulated.  EAP  estimates  were  computed  using 
the  modified  Clark  algorithm  with  R  =  R*  and  R  =  I,  and  by  one  and  two- 
dimensional  quadrature  solutions.  The  results  were  summarized  in  terms  of  the 
mean  and  standard  deviation  of  tfi,  the  root  mean  square  error  (MSB),  and  average 
PSD. 


4.4  Results 

The  results  for  tests  of  length  10,  20,  and  40  items  are  displayed  in  Tables  5-7.  The 
results  are  summarized  as  follows: 

1.  Both  two-dimensional  quadrature  and  the  modified  Clark  algorithm  produce 
extremely  similar  average  ability  estimates  for  10  and  20  item  tests,  at  all  three  ability 
levels,  with  and  without  conditional  dependence.  .At  40  items,  however,  tlie  modifu'd 
Clark  algorithm  produces  a  slight  downward  bias  in  the  average  estiniated  ability, 
which  is  in  turn  re.sponsible  for  increased  .M.SEs  relative  to  the  t wo-diiiu'iisioiial 
quadrature  solution. 

2.  As  expected,  the  variability  in  the  uncorrected  ability  estimates  (i.e.,  one-dimensional 
quadrature  and  Clark  with  R  =  I)  was  consistently  increased  over  the  modified  Clark 
and  two-dimensional  quadrature  solutions,  but  within  each  set  were  virtually  iden¬ 
tical. 
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3.  As  expected,  the  MSE  of  the  uncorrected  ability  estimates  (t.e.,  one- dimensional 
quadrature  and  Clark  with  R  =  I)  was  consistently  increased  over  the  modified 
Clark  and  two-dimensional  quadrature  solutions,  but  within  each  set  were  virtually 
identical.  The  exception  to  this  was  the  previously  noted  result  for  40  items. 

4.  As  expected,  the  average  PSD  of  the  uncorrected  ability  estimates  (he.,  one¬ 
dimensional  quadrature  and  Clark  with  R  =  I)  was  consistently  underestimated 
relative  to  the  modified  Clark  and  two-dimensional  quadrature  solutions,  but  within 
each  set  were  virtually  identical.  This  result  was  consistent  for  all  simulated  condi¬ 
tions. 

5  Estimating  Item  Parameters  in  the  Presence  of  Conditional  De¬ 
pendence 

By  replacing  P{x  =  a:,-  |  di)  with  P(x  =  x,-  |  di,  E),  we  may,  in  theory,  obtain  consis¬ 
tent  estimates  of  item  parameters  and  corresponding  interval  estimates,  legardless 
of  the  level  of  residual  dependence  that  remains  after  we  condition  on  To  do 
this  requires  some  estimate  of  E.  Three  general  approaches  are  available.  First, 
as  in  the  previous  section,  we  may  fit  a  /i -dimensional  model  using  the  methods 
described  by  Bock  and  Aitkin  (1981)  and  Bock,  Gibbons  and  Muraki  (1988).  then 
compute  E  based  on  the  factor  loading  coefficients  obtained  from  factors  2  through 
K.  Second,  we  may  use  the  tetrachoric  correlation  matrix  to  approximate  the  total 
association  in  the  population  and  express  the  residual  association  as  the  difference 
between  the  elements  of  the  tetrachoric  matrix  and  the  expected  correlation  ma¬ 
trix  based  on  provisional  estimates  of  the  one-factor  model  on  each  iteration.  In 
this  way,  residual  covariances  are  computed  from  the  estimated  item  parameters 
on  each  iteration.  Third,  we  may  directly  model  the  elements  of  E,  by  eissuming 
that,  once  we  condition  on  9i,  all  that  remains  is  a  first-order  autocorrelation  among 
the  residual  errors  of  measurement.  This  type  of  structure  might  be  plausible  in 
the  adaptive  testing  situation  in  which  the  presentation  of  future  items  is  based 
on  the  success  of  previous  items.  As  an  illustration  of  this  approach,  we  applied 
the  second  alternative  to  the  well  known  5-item  LSAT  section  7  dataset.  Bock  and 
Lieberman  (1970)  showed  that  these  data  did  not  fit  a  unidimensional  normal  model 
{G^  =  31.59,  <//  =  21, p  <  .05);  however,  Bock  and  Aitkin  were  able  to  fit  a  two- factor 
normal  model  to  these  data  with  reasonable  success  (G^  =  21.23, df  =  17, p  <  .22). 
In  contrast,  the  unidimensional  mode!  of  dependence,  also  fit  these  data  reasonably 
well  (G^  =  21.S3. df  =  16.p  <  .15). 

Inspection  of  the  parameters  estimates  in  Table  8.  for  all  three  models,  reveal 
several  interesting  results. 

1.  The  assumption  of  conditional  independence  is  a  convenience,  but  not  a  re¬ 
quirement  for  estimating  the  parameters  of  IRT  models. 

2.  The  model  of  dependence  identified  a  primary  ability  dimension  represented 
by  all  of  the  items  and  a  few  modest  residual  covariances: 
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V  — 


(25) 


0.71 

-.14 

0.55 

-.13 

-.09 

0.39 

+  .07 

-.08 

-.05 

0.83 

+  .06 

-.14 

-.06 

-.01 

0.83 


In  contrast,  the  2-factor  model,  was  dominated  by  only  2  items,  making  the 
dependence  model  a  more  parsimonious  solution. 


3.  The  improvement  in  fit  of  the  model  of  dependence  over  the  model  of  indepen¬ 
dence  was  substantial. 


4.  Standard  errors  for  the  models  of  dependence  are  slightly  larger  than  for  the 
model  of  independence,  but  considerably  smaller  than  the  2-factor  model.  This 
result  may  suggest  that  the  2-factor  model  is  not  completely  identified  for  this 
5-item  test. 


5.  The  iterative  solution  proposed  here  converged  to  the  same  solution  from  a 
variety  of  starting  values,  which  suggests  that  estimation  of  item  parameters, 
from  which  the  residual  correlations  are  computed,  does  not  lead  to  an  inde¬ 
terminate  solution.  This  is  useful  because  it  avoids  “two-stage”  solutions. 

6.  In  the  present  example,  the  sample  tetrachoric  solution  performed  quite  well. 
.Alternatively,  a  model  of  higher  dimension  could  be  used  to  estimate  the  overall 
sample  correlation  matrix  R.  In  the  present  example,  this  leads  to  a  fit  statistic 
of  Gi7  =  23.96, p  <  .12  as  compared  to  =  21.83,p  <  .15  for  the  tetrachoric 
solution,  which  suggests  that  the  tetrachoric  solution  is  quite  adequate  for  this 
purpose. 


6  Summary  and  Conclusions 

The  results  of  this  research  reveal  that  it  is  quite  possible  to  estimate  accurately  the 
multivariate  normal  orthant  probabilities  for  high-dimensional  normal  integrals  with 
no  restriction  on  the  form  of  the  covariance  matrix  or  mean  vector.  Prior  to  these 
results,  closed  form  solutions  were  only  available  for  the  trivariate  normal  distribution 
and  approximations  of  even  quadrivariate  normal  integrals  were  available  only  for 
special  ca.ses  such  as  equa-correlation  and  “band  matrices"  (Kendall.  1941:  Moran. 
1948;  McFadden.  1960;  .Vbrahamson.  1964:  (.’liilds.  1967:  Dutt.  1973:  and  Dun  ami 
Lin.  1975). 

Application  of  the  Clark  based  method  to  problems  in  item-response  theory  seems 
promising.  Our  results  on  Bayes  estimation  of  ability,  estimating  item  parameters, 
and  testing  goodness  of  fit,  clearly  reveal  that  the  modified  Clark  algorithm  resolves 
typical  cases  of  conditional  dependence  and  provides  a  statistical  solution  that  pre¬ 
serves  the  intended  focus  of  the  test- 
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The  results  of  this  work  have  application  in  other  areas  of  statistics  as  well.  As 
an  example,  the  modified  Clark  algorithm  can  be  used  to  solve  the  problem  of  mul¬ 
tivariate  generalizations  of  probit  analysis  (Ashford  and  Sowden,  1970).  Similarly, 
the  likelihood  of  multinomial  probit  models  used  for  evaluating  discrete  choice  prob¬ 
lems  with  correlated  choice  alternatives  (Daganzo,  1979),  can  also  now  be  evaluated. 
Furthermore,  as  demonstrated  by  Gibbons  and  Bock  (1987),  the  modified  Clark  al¬ 
gorithm  can  also  be  used  to  estimate  the  parameters  of  a  random  effects  probit  model 
in  which  the  errors  of  measurement  exhibit  first-order  autocorrelation. 
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Table  I 


Conditional  Independence 
5  Items 

Probability  Estimates 


pattern 

Monte  Carlo 

Clark 

.Mod.  Clark 

Mod.  Clark 

Quadrature 

Quadrature 

R=I 

1-dimensional 

2-dimensionaJ 

00000 

.0092690 

.0016769 

.0058844 

.0058844 

.0049920 

.0049920 

00001 

.0000310 

.0000358 

.0000166 

.0000166 

.0000270 

.0000270 

00010 

.0002670 

.0003293 

.0001777 

.0001777 

.0002229 

.0002229 

00011 

.0000010 

.0000079 

.0000003 

.0000003 

.0000014 

.0000014 

00100 

.001.5670 

.0010638 

.0013925 

.0013925 

.0012793 

.0012793 

00101 

.0000050 

.0000229 

.0000049 

.0000049 

.0000084 

.0000084 

00110 

.0000840 

.0002130 

.0000427 

.0000427 

.0000701 

.0000701 

01000 

.0082570 

.0022187 

.0080540 

.0080540 

.0069502 

.0069502 

01001 

.0000480 

.0000484 

.0000217 

.0000217 

.0000486 

.0000486 

01010 

.0004120 

.0004384 

.0003050 

.0003050 

.0004077 

.0004077 

01011 

.0000030 

.0000108 

.0000006 

.0000006 

.0000036 

.0000036 

01100 

.0023920 

.0014105 

.0024988 

.0024988 

.0023838 

.0023838 

01101 

.0000250 

.0000311 

.0000077 

.0000077 

.0000217 

.0000217 

OHIO 

.0001780 

.0002845 

.0001162 

.0001162 

.0001846 

.0001846 

Ollll 

.0000030 

.0000072 

,0000003 

.0000003 

.0000022 

.0000022 

10000 

.0908200 

.0708713 

.0869044 

.0869044 

.0858275 

.0858275 

10001 

.0008220 

.0015482 

.0007681 

.0007681 

.0008440 

.0008440 

10010 

.0069930 

.0071212 

.0081681 

.0081681 

.0072004 

.0072004 

10011 

.0001080 

.0004378 

.0000584 

.0000584 

.0000990 

.0000990 

10100 

.0429120 

.0384821 

.0488560 

.0488560 

.0437355 

.0437355 

10101 

.0005770 

.0013366 

.0005361 

.0005361 

.0006260 

.0006260 

10110 

.0051440 

.0048801 

.0056860 

.0056860 

.0054192 

.0054192 

10111 

.0001000 

.0004327 

.0000593 

.0000593 

.0001080 

.0001080 

11000 

.3252630 

.3454604 

.3215043 

.3215043 

.3339557 

.3339557 

11001 

.0058970 

.0103496 

.0068320 

.0068320 

.0059707 

.0059707 

11010 

.0505540 

.05-53977 

.0589594 

.0589594 

.0521171 

.0521171 

11011 

.0013330 

.0034446 

.0013598 

.0013598 

.0013324 

.0013324 

moo 

.3440720 

.3509381 

.3384136 

.3384136 

.3489993 

.3489993 

11101 

.0099590 

.0128232 

.0118505 

.0118505 

.0095529 

.0095529 

11110 

.0891580 

.0840372 

.0879184 

.0879184 

.0845265 

.0845265 

11111 

.0037460 

.0046399 

.0036022 

.0036022 

.0030822 

.0030822 

average 

difference 

.0028954 

.0012275 

.0012275 

.0010744 

.0010744 

maximum  difference 

.0201974 

.0084054 

.0084054 

.0086927 

.0086927 

Table  2 


Conditional  Dependence 
5  Items 

Probabibty  Estimates 


pattern 

Monte  Carlo 

Clark 

Mod.  Clark 

Mod.  Clark 

Quadrature 

Quadrature 

R=I 

1- dimensional 

2-dimension2d 

00000 

.0132430 

.0003325 

.0086805 

.0058537 

.0049920 

.0070033 

00001 

.0000230 

.0000102 

.0000117 

.0000160 

.0000270 

.0000210 

00010 

.0001820 

.0000824 

.0001344 

.0001729 

.0002229 

.0001637 

00100 

.0012020 

.0002234 

.0012086 

.0013689 

.0012793 

.0009430 

00101 

.0000030 

.0000068 

.0000028 

.0000046 

.0000084 

.0000038 

00110 

.0000310 

.0000562 

.0000229 

.0000411 

.0000701 

.0000307 

01000 

.0065430 

.0003425 

.0071056 

.0080097 

.0069502 

.0052413 

01001 

.0000270 

.0000106 

.0000121 

.0000208 

.0000486 

.0000228 

01010 

.0002020 

.0000850 

.0001486 

.0002965 

.0004077 

.0001849 

01011 

.0000010 

.0000027 

.0000002 

.0000006 

.0000036 

.0000011 

01100 

.0011580 

.0002304 

.0013144 

.0024581 

.0023838 

.0010983 

01101 

.0000060 

.0000070 

.0000034 

.0000073 

.0000217 

.0000066 

OHIO 

.0000430 

.0000581 

.0000376 

.0001120 

.0001846 

.0000547 

10000 

.0981450 

.0643565 

.0939096 

.0869296 

.0858278 

.0921583 

lOOOl 

.0006680 

.0009775 

.0006281 

.0007487 

.0008440 

.0006757 

10010 

.0056640 

.0047805 

.0066756 

.0080588 

.0072004 

.0057416 

10011 

.0000580 

.0002493 

.0000434 

.0000560 

.0000990 

.0000635 

10100 

.0355610 

.0282723 

.0421209 

.0488300 

.0437357 

.0362899 

10101 

.0003900 

.00075.58 

.0004003 

.0005209 

.0006260 

.0004223 

10110 

.0034520 

.0026826 

.0038756 

.0055940 

.0054193 

.0036949 

10111 

.0000660 

.0002323 

.0000378 

.0000568 

.0001080 

.0000630 

11000 

.3227360 

.3646366 

.3181484 

.3218323 

.3339569 

.3342406 

11001 

.0051340 

.0094979 

.0062500 

.0067287 

.0059707 

.0053714 

11010 

.0462420 

.0509119 

.0549946 

.0589751 

.0521173 

.0479139 

non 

.0011850 

.0032962 

.0012766 

.0013298 

.0013324 

.0011967 

11100 

.3468730 

.3639200 

.3413597 

.3387589 

.3490006 

.3532162 

11101 

.0103010 

.0124808 

.0120872 

.0117273 

.0095529 

.0098653 

lino 

.0963190 

.0867242 

.0953072 

.0879407 

.0845268 

.0906198 

11111 

.0045420 

.0047778 

.0042022 

.0035504 

.0030822 

.0036917 

average  difference 

.0050750 

.0014122 

.0025535 

.0023840 

.0014398 

maximum  difference 

.0419006 

.0087526 

.0132690 

.0123172 

.0115046 

Table  3 


Conditional  Independence 
10  Items 

Probability  Estimates 


pattern 

Monte  Carlo 

Clark 

Mod.  Clark 

Mod.  Clark 
R=I 

Quadrature 

1-dimensional 

Quadrature 

2-dimensional 

0000001100 

.0000001 

.0000114 

.0000000 

.0000000 

.0000001 

.0000001 

0000100001 

.0000011 

.0000082 

.0000010 

.0000010 

.0000009 

.0000009 

0001010000 

.0000118 

.0001839 

.0000053 

.0000053 

.0000112 

.0000112 

0110010000 

.0001329 

.0006086 

.0000856 

.0000856 

.0001406 

.0001406 

0110100000 

.0016666 

.0013821 

.0012122 

.0012122 

.0014266 

.0014266 

1001100000 

.0027151 

.0042885 

.0019134 

.0019134 

.0024568 

.0024568 

lOlOOOOCOO 

.0067213 

.0079889 

.0055955 

.0055955 

.0058672 

.0058672 

1011000000 

.0089667 

.0111335 

.0085417 

.0085417 

.0085844 

.0085844 

1100000000 

.0146445 

.0186212 

.0131084 

.0131084 

.0134876 

.0134876 

1101000000 

.0220419 

.0296640 

.0227618 

.0227618 

.0218673 

.0218673 

1110000000 

.0462991 

.0617006 

.0463947 

.0463947 

.0466949 

.0466949 

1110100000 

.0477081 

.0625243 

.0501873 

.0501873 

.0490546 

.0490546 

1111000000 

.0968781 

.1280707 

.0959419 

.0959419 

.0998421 

.0998421 

1111100000 

.1177844 

.1480838 

.1172587 

.1172587 

.1223589 

.1223589 

average  difference 

.0078048 

.0006538 

.0006.538 

.0008825 

.0008825 

all  patterns 

.0007246 

.0001281 

.0001281 

.0000643 

.0000643 

maximum 

difference 

.0311926 

.0024792 

.0024792 

.0045745 

.0045745 

all  patterns 

.0311926 

.0066958 

.0066958 

.0045745 

.0045745 

Table  4 


Conditional  Dependence 
10  Items 

Probability  Estimates 


pattern 

Monte  Carlo 

Clark 

.Mod.  Clark 

Mod.  Clark 
R=I 

Quadrature 

1-dimensional 

Quadrature 

2-dimensional 

0000001100 

.0000001 

.0000159 

.0000000 

.0000000 

.0000003 

.0000000 

0000100001 

.0000009 

.0000046 

.0000008 

.0000010 

.0000009 

.0000009 

0001010000 

.0000023 

.0000667 

000021 

.0000053 

.0000112 

.0000026 

0110010000 

.0000328 

.0001205 

.0000301 

.0000856 

.0001406 

.0000337 

0110100000 

.0013382 

.0002668 

.0010167 

.0012122 

.0014266 

.0011223 

1001100000 

.0028129 

.0033192 

.0020223 

.0019134 

.0024568 

.0025560 

1010000000 

.0077306 

.006.5970 

.0064644 

.0055955 

.0058672 

.0068002 

iOllOOCOOO 

.0088950 

.0078580 

.0077004 

.0085417 

.0085844 

.0084590 

1 100000000 

.0176867 

.0209829 

.0162748 

.0131084 

.0134876 

.0164043 

1101000000 

.0240336 

.0.305718 

.0246059 

.0227618 

.0218673 

.0238656 

11 10000000 

.0517423 

.0713755 

.0521337 

.0463947 

.0466949 

.0523689 

1110100000 

.0506626 

.0678296 

.0527083 

.0501873 

.0490546 

.0523895 

1111000000 

.1033206 

.1465492 

.1029978 

.0959419 

.0998421 

.1073350 

111  1100000 

.1212253 

.164.5829 

.1208567 

.1172587 

.1223589 

.1274467 

average  difference 

.00979.58 

.0006206 

.0018991 

.0014549 

.0011343 

all  patterns 

.0008165 

.0001337 

.0002247 

.0002245 

.0000912 

maximum 

difference 

.0433576 

.0020457 

.0073787 

.0050474 

.0062214 

all  patterns 

.0433576 

.0059526 

.0117328 

.0116041 

.0062214 

Table  5 


Results  of  Simulation 
EAR  Ability  Estimates 
10  Items 


Condition 

Mean 

SD 

MSE 

PSD 

Independence 

Quadrature 

-.0123 

.2694 

.2697 

.4670 

Clark 

-.0165 

.2768 

.2772 

.4624 

0  =  1 

Quadrature 

.7876 

.2415 

.3218 

.4557 

Clark 

.7831 

.2498 

.3309 

.4483 

9  =  2 

Quadrature 

1.5720 

.1582 

.4566 

.4179 

Clark 

1.5692 

.1652 

.4618 

.4150 

Dependence 

9  =  0 

Quadrature  ID 

-.0189 

.3044 

.3050 

.4586 

Clark  Rsl 

-.0239 

.3148 

.3157 

.4525 

Quadrature  2D 

-.0154 

.2638 

.2643 

.4775 

Modified  Clark 

-.0191 

.2674 

.2681 

.4765 

9  =  1 

Quadrature  ID 

.7974 

.2665 

.3365 

.4316 

Clark  R=I 

.7915 

.2769 

.3467 

.4437 

Quadrature  2D 

.7753 

.2255 

.3185 

.4648 

Modified  Clark 

.7703 

.2291 

.3246 

.4609 

9  =  2 

Quadrature  ID 

1.5754 

.1699 

.45T8 

.4174 

Clark  R=l 

1.5735 

.1785 

.4628 

.4143 

Quadrature  2D 

1.5560 

.1432 

.4669 

.4185 

Modified  Clark 

1.5568 

.1507 

.4685 

.4173 

Table  6 


Results  of  Simulation 
EAR  Ability  Estimates 
20  Items 


Condition 

Mean 

SD 

MSE 

PSD 

Independence 

&  =  0 

Quadrature 

.0125 

.2881 

.2883 

.4200 

Clark 

-.0244 

.2953 

.2963 

4231 

e  =  1 

Quadrature 

.8285 

.2784 

.3271 

.4147 

Clark 

.7934 

.2791 

.3474 

.4132 

9  =  2 

Quadrature 

1.6600 

.2414 

.4172 

.3992 

Clark 

!  Dependence 

\ 

1.6434 

.2543 

.4383 

.3926 

if 

o 

Quadrature  ID 

-.0104 

.3588 

.3589 

.4036 

Clark  R=I 

-.0366 

.3641 

.3659 

.4052 

Quadrature  2D 

-.0070 

.2766 

.2767 

.4551 

Modified  Clark 

-.0359 

.2799 

.2822 

.4642 

9  =  1 

Quadrature  ID 

.8222 

.3385 

3824 

.4004 

Clark  R=I 

.7879 

.3374 

.3986 

.3966 

Quadrature  2D 

.7819 

2522 

.3336 

4418 

Modified  Clark 

.7445 

.2525 

.3594 

.4410 

9  =  2 

Quadrature  ID 

1.6745 

.2879 

.4348 

.3933 

Clark  R=1 

1.6585 

.2971 

.4529 

.3876 

Quadrature  2D 

1.6077 

.2092 

.4449 

.4095 

Modified  Clark 

1.5932 

.2193 

.4625 

.4095 

Table  7 


Results  of  Simulation 
EAP  Ability  Estimates 
40  Items 


Condition 

Mean 

SD 

MSE 

PSD 

Independence 

0  =  0 

Quadrature 

.0025 

.2985 

.2985 

.3515 

Clark 

-.0845 

.3336 

.3442 

.3595 

e  =  1 

Quadrature 

.8730 

.2927 

.3191 

.3521 

Clark 

.7809 

.3375 

.4024 

0  =  2 

Quadrature 

1.7406 

.2664 

.3720 

.3565 

Clark 

1.6729 

.2990 

.4434 

1 

.3525 

Dependence 

0  =  0 

Quadrature  ID 

-.0354 

.4224 

.4239 

.3285 

Clark  R=I 

-.1038 

.4123 

.4252 

.3392 

Quadrature  2D 

-.0254 

.2906 

.2917 

Modified  Clark 

-.1318 

.3230 

.3489 

11 

0  =  1 

Quadrature  ID 

.8612 

.4229 

.4452 

3266 

Clark  R=1 

.7786 

.4195 

.4744 

..'273 

Quadrature  2D 

.7909 

.2851 

.3537 

.4i75 

Meidified  Clark 

6602 

.3129 

.4622 

.3991 

0  =  2 

Quadrature  ID 

1 .7.585 

.3850 

.4546 

..3369 

Clark  R=1 

1.6973 

.3973 

.4997 

..33.36 

Quadrature  2D 

1.6272 

.2428 

.4453 

.3917 

Modified  Clark 

1.5462 

.2671 

.5269 

.3883 
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